UNITED  STATES  AIR  FORCE 
AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 

Wright-Patterson  Air  Force  Base, Ohio 


DiSTPjBUTipNj 
Appioved  toi  P 
Distributt** 


I IKISSIM  It 


fits 

W»  it  Sc  - 

MC 

1,1  SiUlM 

mu. 

Ain-  .C»IU» 

1 

IT 

i 

| iiSTitiuTioe  miuiiuir  cocrs  ■ 

— 

1 tut 

urn  m3  af  Sitcni  ' 

£ 

1 

| 1 

A PARAMETRIC  COSTING  MODEL  FOR 
FLIGHT  SIMULATOR  ACQUISITION 

Milton  C.  Ross,  Captain,  USAF 
Gerald  L.  Yarger,  Captain,  USAF 

SLSR  22-76B 


D D C 

ln)r?frorpnn  pp 

NOV  23  1976 

tetsEircrElli) 

B 


ffl 

51 


OlSiT:  .:ON  STATF^ETfl^.?  1 

Approved  for  public  relecse;  I 

Distribution  Uniumted  I 


USAF  SCN  7S-20R  AFIT  Cnnrrol  Number  SLSR  22-76B 

AFIT  RI:Sr-\KC!  .\SSHSS'-!1'.\T 

The  purpose  of  this  questionnaire  is  to  determine  the  potential  for  current 
and  future  applications  of  AFIT  tiics : ^ research.  Please  return  completed 
questionnaires  to:  AFiT/SLC.R  (Thesis  Feedback),  Wright -Patterson  ATB, 

Ohio  43433. 

1.  Did  this  research  contribute  to  a current  Air  Force  project? 

a.  Yes  b.  No 

2.  Do  you  believe  this  research  tonic  is  significant  enough  that  it  would 
have  been  researched  (or  contracted)  by  your  organization  or  another  agency 
if  AFIT  had  not  researched  it? 

a.  Yes  b.  No 

3.  The  benefits  of  AFIT  research  can  often  be  expressed  by  the  equivalent 
value  that  your  agency  received  by  virtue  of  AFIT  performing  the  research. 

Can  you  estimate  what  this  research  would  have  cost  if  it  had  been 
accomplished  under  contract  or  if  it  had  been  done  in-house  in  terms  of  man- 
power and/or  dollars? 

a.  Man-years  $ (Contract) . 

b.  Man-years $ (In-house). 

4.  Often  it  is  not  possible  to  attach  equivalent  dollar  values  to  research, 
although  the  results  of  the  research  may,  in  fact,  be  important.  Whether  or 
not  you  were  able  to  establish  an  equivalent  value  for  this  research  (3  above) , 
what  is  your  estimate  of  its  significance? 

a.  Highly  b.  Significant  c.  Slightly  d.  Of  No 

Significant  Significant  Significance 

5.  Conraents: 


Name  and  Grade 


Position 


Organization 


Location 


UNCLASSIFIED 

SECURITY  CLASSIFICATION  OF  This  pace  2HTi«n  P»l«  gnur.rf) 

f REPORT  DOCUMENTATION  PAGE 


ae-P.QHr  imam ■— 

SLSR-22-76B 


3App  READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM 

2.  GOVT  ACCESSION  NO.  3 RECIPIENT'S  CATALOG  NUMBER 


«.  title  (•«"<«  Subtitle! 

£ PARAMETRIC  COSTING  MODEL  FOR 
FLIGHT  SIMULATOR  ACQUISITION. 


K 5 TYPE  OF  REPORT  & PERIOD  COVEREO 

V Master's  Thesis, 


7.  AuTHQflf  a.  CONTRACT  OR  GRANT  NUMBERS.) 

Milton  C./ross^  Captain,  USAF 
Gerald  L./Yarger j Captain,  USAF 

1 performing  organization  name  and  address  »0  program  element,  project,  task 

AREA  h WORK  UNIT  NUMBERS 

Graduate  Education  Division  ___  - 

School  of  Systems  and  Logistics  / ) 

Air  Force  Institute  of  Technology, WPAFB ,()hi'//V 

II.  CONTROLLING  OFFICE  NAME  ANO  AOORESS  l2.MEP.QgX DAXE  .... 

Department  of  Research  and  Communicative  /SeplWtetiswr  1976  | 
Studies  ( SLGR)  i^-nuiSSfR  b*  Images 

AFIT/SLGR,  WPAFB,  OH  45433 55 

14  MONITORING  AGCMCY  N AM  E A AGO  R ESS(//  dllteren  f from  Controlling  Office,  15.  SECURITY  CLASS,  fol  thle  report) 

V ~ UNCLASSIFIED 


15*  DECL  ASSI  FI  CATION/ OOWNGRAOING 
SCHEDULE 


MV  DISTRIBUTION  STATEMENT  (ot  thle  Report) 


Approved  for  public  release;  distribution  unlimited 


\V7. DISTRIBUTION  STATEMENT  (ot  the  abetract  entered  In  Block  20,  II  different  from  Report) 


18.  SUPPLEMENTARY  NOTES 


APPROVED  Jp^jPUBLIC  RELEASE  APR  I9C  -17. 
JEIfCiF.  GUESS,  CAFT,  USAF 


M9  key  WORDS  ( Continue  on  reveree  eide  It  neceeemry  and  Identify  by  bloc*  number) 


parametric  model,  flight  simulator  acquisition,  cost  estimating 
relationship,  cost  estimation 


20  ABSTRACT  ( Continue  on  reveree  eide  It  neceeemry  and  Identify  by  block  number) 

Thesis  Chairman:  Eugene  E.  Jones,  Major,  USAF 


DD  , :°nrm7,  1473  EDITION  OF  I NOV  SS  IS  OBSOLETE 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  CRT) mn  Dala.Enltrrd} 

Oil 


SECURITY  CLASSIFICATION  OF  This  PAGE(»?i«r  Dmta  Enlertd) 


The  acquisition  of  flight  simulators  has  been 
plagued  by  cost  growth  difficulties  arising  from 
poor  initial  cost  estimates.  A parametric  costing 
model  for  simulator  acquisition  was  perceived  as  a 
tool  which  could  provide  required  cost  estimates  early 
in  the  acquisition  process.  This  research  effort 
utilized  multiple  regression  analysis  to  formulate  a 
parametric  costing  model  or  cost  estimating  relation- 
ship (CER)  for  specific  application  to  flight  simulator 
acquisition.  Data  derived  from  existing  simulator 
systems  which  pertained  to  system  cost  as  well  as 
physical  and  performance  characteristics  provided  the 
basis  for  model  development.  A significant  relation- 
ship was  found  to  exist  between  cost  and  several  system 
characteristics.  This  relationship  was  validated 
through  various  statistical  tests  and  enabled  model 
formulation.  The  research  indicated  that  continuing 
collection  and  utilization  of  CER  data  would  provide 
for  further  refinement  of  the  parametric  costing  model. 


SLSR  22-1 6b 


A PARAMETRIC  COSTING  MODEL  FOR 
FLIGHT  SIMULATOR  ACQUISITION 


A Thesis 

Presented  to  the  Faculty  of  the  School  of  Systems  and  Logistics 
of  the  Air  Force  Institute  of  Technology 
Air  University 

In  Partial  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  of  Science  in  Logistics  Management 


By 


Milton  C.  Ross,  BS 
Captain,  USAF 


Gerald  L.  Yarger,  BS 
Captain,  USAF 


September  1976 


Approved  for  public  release 
distribution  unlimited 


L 


ABSTRACT 


The  acquisition  of  flight  simulators  has  been 
plagued  by  cost  growth  difficulties  arising  from  poor 
initial  cost  estimates.  A parametric  costing  model  for 
simulator  acquisition  was  perceived  as  a tool  which  could 
provide  required  cost  estimates  early  in  the  acquisition 
process.  This  research  effort  utilized  multiple  regression 
analysis  to  formulate  a parametric  costing  model  or  cost 
estimating  relationship  (CER)  for  specific  application  to 
flight  simulator  acquisition.  "~Bata  derived  from  existing 
simulator  systems  which  pertained  to  system  cost  as  well 
as  physical  and  performance  characteristics  provided  the 
basis  for  model  development.^  A significant  relationship 
was  found  to  exist  between  cost  and  several  system  charac- 
teristics. This  relationship  was  validated  through  various 
statistical  tests  and  enabled  model  formulation.  The 
research  indicated  that  continuing  collection  and  utiliza- 
tion of  CER  data  would  provide  for  further  refinement  of 
the  parametric  costing  model. 
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Chapter  1 


BACKGROUND 

STATEMENT  OF  THE  PROBLEM 

Military  acquisition  and  use  of  flight  simulators 
has  been  growing  rapidly  due  to  the  Department  of  Defense 
(DOD)  goal  of  reducing  total  military  flying  hours  by  25 
percent  prior  to  1980  (24:2).  DOD  has  committed  its  com- 
ponent arms  to  the  increased  use  of  flight  simulators. 
Funds  allocated  to  simulator  acquisition  have  increased 
significantly  during  the  past  three  years  (24:3).  Simu- 
lator acquisitions  have  been  documented  which  experienced 
both  large  cost  overruns  and  system  performance  below 
specifications  due  to  underfunding.  Underfunding  has 
resulted  from  inaccurate  cost  estimates  (9:42).  The 
flight  simulator  acquisition  process  has  not  utilized  the 
valid  and  accepted  parametric  method  of  cost  estimation 
(6) . A tested  parametric  costing  model  (as  defined  later 
in  this  chapter)  for  flight  simulator  acquisition  does  not 
exist  at  present. 

JUSTIFICATION  AND  DELIMITATION 

The  DOD  flying  hour  reduction  goal  has  been  estab- 
lished due  to  the  forecast  scarcity  of  fuels  as  well  as 
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federal  budgetary  constraints  (24:25).  To  fill  the  gap 
created  by  reduced  flying,  the  military  services  have  accel- 
erated the  acquisition  and  employment  of  flight  simulators 
and  other  training  devices  (3:7).  The  dramatic  increase  in 
the  importance  of  simulators  is  also  demonstrated  by  the 
increase  in  funds  allocated  to  them.  Funding  to  update 
existing  training  devices  and  initiate  acquisition  programs 
for  other  systems  has  increased  from  $88.5  million  for  fis- 
cal year  1974  to  $283  million  for  fiscal  year  1975  (24:24). 
Since  acquisitions  of  this  magnitude  attract  great  public 
and  congressional  scrutiny,  it  behooves  acquisition  managers 
to  function  as  efficiently  as  possible.  In  the  past  this 
has  not  been  the  case  as  the  following  examples  demonstrate. 

Cost  overruns  have  been  documented  on  two  flight 
simulator  systems,  the  F-111A  and  the  A-7E.  These  overruns 
have  totalled  $4.2  million  to  date  (9:38;  3:62).  During  the 
acquisition  of  the  F-111A  simulator 

. . . cost  considerations  adversely  affected  the 
performance  and  delivery  data  of  the  final  product. 
System  performance  was  degraded  by  lack  of  funding 
and  the  funding  provisions.  System  performance  was 
degraded  to  the  point  of  negative  training  [9:42], 

As  has  been  the  experience  with  other  recently  pro- 
cured DOD  systems  and  material,  the  problem  of  cost  overruns 
in  flight  simulator  acquisition  has  evolved  primarily  from 
unrealistically  low  initial  cost  estimates  as  well  as  from 
changing  specifications  (7:3).  A direct  consequence  of 
those  cost  overruns  related  to  underfunding  has  been  sub- 
optimal  system  effectiveness.  Less  directly,  cost  overruns 
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have  served  to  diminish  public  confidence  in  the  managerial 
skill  of  its  military  leaders. 

The  need  for  accurate  cost  estimates  may  be  over- 
shadowed by  the  need  for  timely  cost  estimates.  If  accurate 
cost  estimates  can  be  furnished  early  in  the  conceptual 
phase  of  acquisition,  these  cost  estimates  will  aid  in 
several  decision  situations.  Identifying  possible  cost/per- 
formance trade-offs  in  the  design  effort,  providing  a basis 
for  cost-effectiveness  reviews,  and  providing  information 
useful  in  ranking  known  alternatives  are  among  these  recur- 
ring situations  (12:6).  As  one  researcher  has  noted: 
"Parametric  Cost  Estimates  can  provide  estimates  during  the 
conceptual  formulation  stage  of  the  acquisition  process 
before  detailed  engineering  plans  are  available  [12:6].'' 

The  acquisition  of  flight  simulators  has  been  based 
exclusively  on  industrial  engineering  cost  estimation  (1:1). 

At  this  writing  however  an  attempt  is  under  way  to  validate 
a simulation  model  for  the  estimation  of  flight  simulator 
acquisition  cost  (25) . As  has  been  noted  previously,  cost 
estimates  for  simulator  acquisition  have  fallen  short  of 
expectations  in  that  cost  overruns/underfunding  have  resulted. 
Thus  far,  parametric  costing  procedures  have  not  been  uti- 
lized in  the  acquisition  of  flight  simulators  (1:2). 

The  traditional  industrial  engineering  approach  to 


cost  estimating  relies  on  detailed  cost  analysis  of  human 
and  material  input  to  the  system  under  development.  The 
parametric  costing  procedure  utilizes  historical  cost  data 
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derived  from  existing  similar  systems.  The  industrial 
engineering  approach  fails  to  account  for  engineering  and 
design  specification  changes  and  other  items  that  are  not 
identifiable  at  the  time  of  design  (7:6).  Because  the  para- 
metric estimating  procedure  can  account  for  unforeseen  dif- 
ficulties prior  to  their  occurrence  during  development,  it 
is  by  far  the  preferred  procedure  for  cost  estimation  early 
in  the  acquisition  process.  As  the  system  becomes  more 
clearly  defined  the  industrial  engineering  approach  becomes 
increasingly  accurate.  At  some  undetermined  and  variable 
point  in  the  acquisition  process  the  utility  of  the  indus- 
trial engineering  approach  will  exceed  the  utility  of  the 
parametric  approach  due  to  the  greater  accuracy  of  the 
industrial  engineering  estimate  (21:3). 

In  summary,  the  continual  evaluation  of  aerospace 
systems  provides  a serious  challenge  to  USAF  acquisition 
managers.  This  challenge  pertains  not  only  to  primary 
flight  systems  but  also  to  associated  flight  system  simu- 
lators. The  absence  of  accurate  initial  cost  estimates  is 
central  to  this  problem.  Creation  of  a method  to  provide 
accurate  cost  estimates  early  in  the  acquisition  cycle 
would  significantly  aid  USAF  managers.  Captain  James  H. 
Coile,  Plans  and  Programs  Officer,  Systems  Program  Office 
(SPO)  for  USAF  Flight  Simulator  Acquisition,  has  specifi- 
cally recommended  research  in  simulator  cost  estimation 
due  to  his  conviction  that  the  simulator  acquisition  pro- 
gram would  benefit  greatly  from  such  input  (6) . 
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Flight  Simulator  Technology 

The  technology  of  the  nineteen-seventies  has  enabled 
flight  simulators  to  duplicate  all  flight  performance  charac- 
teristics and  yield  an  accurate  visual  representation  of 
flight  (22:29).  Simulator  systems  which  fully  duplicate 
aircraft  characteristics  and  provide  real  three  dimensional 
motion  are  currently  under  development  and  production. 

Through  complex  hydraulic  systems,  modern  simulators  are 
able  to  provide  the  physical  sensations  of  flight  including 
positive  and  negative  gravitational  forces  (17:40).  In 
addition,  the  current  generation  of  digital  computers,  which 
are  the  heart  and  brain  of  modern  flight  simulators,  are 
capable  of  processing  2.5  million  instructions  per  second. 
Technical  sophistication,  through  computer  and  television 
application,  has  evolved  to  the  level  wherein  several  indi- 
vidual simulators  can  be  linked  to  allow  simulation  of 
air-to-air  combat  among  pilots  occupying  separate  simulator 
stations  (14:78) . 

The  rapid  increase  of  flight  simulator  technological 
sophistication  has  logically  generated  rapid  growth  in  costs. 
The  current  generation  of  flight  simulators  has  become  as 
expensive  to  acquire  as  the  aircraft  they  are  designed  to 
simulate  (18:16).  Therefore,  flight  simulator  acquisition 
requires  careful  planning  and  close,  sensitive  attention  to 
cost  factors. 

No  research  has  been  documented  which  addresses 


the  identification  of  those  cost  factors  which  most 
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significantly  affect  total  system  cost  (6). 


Certain  factors 


are  postulated  by  experts  to  relate  most  directly  to  total 
system  cost.  These  factors  include:  unit  weight,  electrical 

power  consumption,  computer  core  capacity,  computer  instruc- 
tion processing  speed,  the  number  of  crew  stations,  the 
number  of  motion  axes,  system  cooling  capacity,  the  number 
of  emergency  situations  simulated  and  the  total  number  of 
sensory  cues  (6). 


Cost  Analysis  Procedures 

Cost  analysis  is  an  integral  part  of  the  acquisition 
process  and  is  a function  in  which  every  procurement  activ- 
ity must  engage.  General  George  S.  Brown,  USAF,  (5:9)  has 
pointed  to  the  significance  of  the  cost  analysis  process  and 
specifically  to  the  origination  of  accurate  and  credible  cost 
estimates.  Recent  trends  in  DOD  management  policy  reflect 
the  importance  of  the  cost  analysis  function.  The  establish- 
ment of  the  Cost  Analysis  Improvement  Group  (CAIG)  in  1972 
and  the  recent  distribution  of  Air  Force  Regulation  173-2, 
USAF  CER/Cost  Factors  Program,  are  evidence  of  the  importance 
placed  on  cost  analysis  by  the  DOD  (7:8;  19:25). 

Early  in  1970  the  Assistant  Secretary  of  Defense  for 
Systems  Analysis  distributed  a memorandum  clarifying  the 
department's  position  on  techniques  to  be  used  in  estimating 
the  cost  of  new  weapons  acquisition.  Cost  estimates  were  to 
be  derived  either  from  detailed  "grass  roots"  calculations 
(the  industrial  engineering  method)  or  based  upon  relation- 
ships between  aggregate  components  of  system  cost  and  the 
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physical  and/or  performance  characteristics  of  the  system. 
These  relationships  were  to  be  derived  from  the  cost  his- 
tories of  prior  programs.  The  latter  of  these  two  accepted 
procedures  is  based  upon  the  parametric  method  (7:2). 

To  assess  the  advantages  and  disadvantages  of  the 
parametric  approach  to  cost  estimation,  it  is  first  neces- 
sary to  discuss  industrial  engineering  cost  estimating 
procedures.  Contrasting  these  two  established  procedures 
thereby  provides  the  framework  for  understanding  their 
respective  applications. 

The  industrial  engineering  method.  In  the  past,  the  defense 
system  cost  estimates  reviewed  by  the  Defense  System  Acqui- 
sition Review  Council  (DSARC)  have  most  frequently  been 
based  on  contractor  estimates  which,  in  turn,  relied  almost 
exclusively  on  the  industrial  engineering,  or  traditional, 
estimating  approach  (21:3).  This  approach  is  also  termed 
the  "grass  roots"  approach  because  it  is  founded  in  the 
exercise  of  gathering  all  the  costs  associated  with  those 
elements  which  combine  to  form  a final  product.  The  summa- 
tion of  these  costs,  which  are  located  at  the  "grass  roots" 
of  the  design,  development  and  production  processes,  pro- 
vides the  sought-after  cost  estimate  (21:3). 

Certain  facts  must  be  known  to  derive  a cost  esti- 
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mate  utilizing  the  industrial  engineering  approach. 
Generally,  the  costs  associated  with  all  the  labor,  material 
and  overhead  which  contribute  to  the  final  product  must  be 
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available  in  a form  such  that  total  costs  can  be  calculated 
based  on  estimates  of  the  input  variables.  The  industrial 
engineering  cost  estimate,  then,  is  a function  of  known  or 
predetermined  costs  of  the  production  input  variables  (11:5). 

Due  to  its  reliance  on  extensive  system  descriptions 
and  design  along  with  standards  built  from  time  and  motion 
studies,  vendor  quotes  and  so  forth,  the  industrial  engi- 
neering approach  is  both  cumbersome  and  expensive  to  use 
(11:4).  Because  the  level  of  detail  required  in  the  indus- 
trial engineering  approach  is  great,  the  time  which  must  be 
devoted  to  analysis  of  those  details  is  also  great.  There- 
fore, expense  and  time  consumption  detract  from  the  effec- 
tiveness of  the  industrial  engineering  approach  (7:26). 

Perhaps  the  greatest  limitation  of  the  industrial  engineer- 
ing approach  is  that  it  has  little  or  no  utility  under 
conditions  of:  poor  or  incomplete  system  definition,  rapid 

and  frequent  changes  in  the  technological  aspects  of  the 
system,  or  changing  of  the  required  operational  capabil- 
ities (7:26)  . 

The  major  strength  of  the  industrial  engineering 
method  is  the  accuracy  of  the  estimate  it  provides  (7:26). 

However,  to  be  accurate  this  method  must  have  access  to 
specific  information  and  the  information,  or  data,  must 
accurately  depict  all  the  factors  which  contribute  to  total 
system  cost.  Reliance  on  data  is,  of  course,  at  the  heart 
of  any  cost  estimating  procedure.  Former  Assistant 
Secretary  of  Defense  for  Systems  Analysis,  Dr.  Alain  Enthover, 
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has  noted  that  successful  cost  analysis  is  characterized  by 
good  data,  good  people,  good  techniques  and  prompt  response. 
He  further  notes  that,  "...  without  good  data , all  the 
people,  techniques  and  speed  are  useless  to  the  analytical 
effort  [2:24]  . " 

The  parametric  method.  Parametric  estimates  predict  costs 
by  means  of  explanatory  variables  such  as  performance 
characteristics,  physical  characteristics  and/or  character- 
istics relevant  to  the  development  process  (e.g.,  milestones) 
which  are  derived  from  experience  on  related  systems  (12:2). 
These  relationships  are  in  the  form  of  empirical  formulas 
that  relate  some  characteristic  of  the  system  to  the  cost 
of  the  same  characteristic  in  a previously  acquired  and 
related  system.  Accordingly,  parametric  estimates  reflect 
delays,  mistakes  and  changing  requirements  based  upon 
actual  past  performance  (21:3). 

The  parametric  approach  can  be  faulted  on  several 
counts  as  can  the  industrial  engineering  method.  The  major 
difficulties  arising  from  the  parametric  approach  to  cost 
estimation  are: 

1.  The  system  for  which  the  estimate  is  being  made 
must  be  closely  related  to  the  system  used  in  the 
data  base  to  define  the  relationship,  and  it  must 
be  assumed  that  the  defined  relationship  is  still 
valid  (12:4)  . 

2.  Accuracy  of  the  cost  estimate  is  limited  by  the 
fact  that  physical  and  performance  characteristics 


10 


do  not  represent  all  of  the  reasons  for  variance  in 
cost  (10:9) . 

3.  When  extrapolating  beyond  the  values  of  data  in 
the  data  base,  confidence  in  the  accuracy  of  the 
estimate  decreases  (12:56). 

4.  The  historical  data  needed  to  define  the  cost 
estimating  relationship  (CER)  may  be  either  expen- 
sive to  obtain  or  nonexistent  (7:26). 

These  deficiencies  in  the  parametric  approach  are  not  uni- 
versal; they  apply  only  to  isolated  situations.  These 
situations  are  all  related  to  the  nonavailability  or  inappro- 
priateness of  the  data  on  which  they  rest.  However,  many 
situations  are  quite  amenable  to  parametric  application. 

Assuming  adequacy  of  data,  the  parametric  method's 
strengths  are: 

1.  Usage  of  actual  past  experience  as  a data  base 
incorporates  setbacks  such  as  engineering  and  design 
specification  changes  that  are  not  identifiable  at 
the  time  of  design  (12:6). 

2.  Cost  estimation  is  possible  early  in  the  con- 
ceptual stage  prior  to  the  completion  of  detailed 
engineering  plans  (12:6). 

3.  Cost  estimates  can  be  made  quickly  and  inexpen- 
sively once  the  relationship  has  been  defined  (7:26). 

4.  The  cost  estimate  is  independent  of  estimator 


bias  and,  hence,  more  likely  to  be  objective  (7:26). 
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These  strengths  of  the  parametric  method  do  not  prove  it  to 
be  the  superior  method  for  cost  estimation.  However,  recent 
experience  at  the  DSARC  level  of  review  has  shown  that  the 
parametric  estimates  submitted  by  the  CAIG  have  been  more 
accurate  than  the  industrial  engineering  estimates  provided 
by  the  services  (7:8). 

The  parametric  and  industrial  engineering  methods  are 
most  effective  when  applied  simultaneously  in  the  acquisition 
process  since  the  strengths  of  each  offset  the  weaknesses  of 
the  other  (12:6).  Currently,  however,  no  parametric  cost 
estimation  model  exists  for  flight  simulator  acquisition. 

OBJECTIVES 

The  primary  objective  of  this  research  was  to  for- 
mulate a parametric  cost  estimating  model  for  specific  use 
in  flight  simulator  acquisition.  Such  a model  was  to  pro- 
vide cost  estimates  throughout  the  acquisition  cycle  but 
was  to  be  based  upon  characteristics  that  are  established 
early  in  the  conceptual  stage  of  acquisition.  In  order  to 
substantiate  the  strength  of  the  model  two  additional 
objectives  were  pursued.  First,  the  model  was  tested  to 
ascertain  its  accuracy  as  a predictor  of  actual  first  unit 
cost  of  a flight  simulator  system.  Second,  the  model  was 
assessed  in  regard  to  its  usefulness  as  a cost  estimation 


tool . 
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RESEARCH  HYPOTHESIS 


There  is  some  combination  of  the  following  flight 
simulator  characteristics  which  have  a significant  relation- 
ship to  simulator  first  unit  cost.  The  characteristics,  all 
of  which  can  be  identified  in  the  conceptual  stage  of  the 
weapon  acquisition  process,  are: 

1.  computer  core  capacity 

2.  computer  instruction  processing  speed 

3.  the  number  of  crew  stations 

4.  motion  axes 

5.  emergency  procedure  capability 

6.  sensory  cues 

7.  unit  weight 

8.  rate  of  electrical  power  consumption. 


Chapter  2 
METHODOLOGY 

OVERVIEW  OF  THE  DATA  PRODUCING  SITUATION 

The  data  of  concern  in  this  statistical  model  build- 
ing research  were  described  by  two  general  definitions. 

Data  which  represented  first  unit  cost  of  various  flight 
simulator  systems  was  one  of  the  classifications.  The  other 
classification  consisted  of  those  physical  characteristics 
of  the  flight  simulator  systems  which  were  believed  to  be 
related  to  first  unit  costs.  Examination  of  all  flight 
simulator  systems  was  neither  feasible  nor  meaningful.  A 
representative  population  of  the  current  generation  of 
flight  simulators  was  utilized. 

Population 

The  population  of  interest  consisted  of  USAF  flight 
simulators  in  operational  use  during  1975.  Included  among 
those  were  systems  which  simulated  transport,  bomber, 
trainer,  fighter  and  special  use  aircraft.  Those  systems 
provided  the  data  for  the  cost  model  development. 

Term  Definition 

1.  First  Unit  Cost  - This  was  the  cost  in  adjusted 
dollars  paid  by  the  USAF  for  the  first  operationally 
installed  unit  of  flight  simulator  systems. 
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2.  Computer  Core  Capacity  - The  maximum  characters 
which  could  be  stored  in  memory. 

3.  Computer  Instruction  Processing  Speed  - The 
internal  speed  of  transmitting  information  to  and/or 
from  memory.  Speed  was  measured  in  microseconds 

(10  ^ seconds) . 

4.  Number  of  Crew  Stations  - The  number  of  physical 
locations  in  the  simulator  system  which  could  be 
manned  by  flight  crew  members. 

5.  Degrees  of  Freedom  - The  number  of  motion  axes 
or  motion  planes  available. 

6.  Sensory  Cues  - The  number  of  general  flight  or 
aircraft  sensations  which  could  be  perceived  through 
either  sight,  hearing  or  sense  of  touch. 

7.  Weight  - The  weight,  in  pounds,  of  the  simulator 
crew  station  including  motion  platform. 

8.  Rate  of  Power  Consumption  - Kilowatts  of 
electricity/hour  required  to  maintain  normal  simu- 
lator operation. 

9.  Emergency  Procedures  - The  total  number  of 
emergency  procedures  and  malfunctions  simulatable. 

10.  Cooling  Capacity  - The  cooling  capacity 
required  for  one  mission  simulator  expressed  in 
BTU/hr . 

Data  and  Data  Sources 

It  was  initially  assumed  that  the  data  specifying 


total  system  costs  and  describing  simulator  system 
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characteristics  was  recorded  in  the  various  individual  flight 
simulator  contract  files  (6).  Examination  of  the  files  was 
to  have  permitted  extraction  of  those  data  values  which  were 
the  foundation  for  the  model  development.  The  data  values 
which  were  used  in  the  model  development  process  were 
ungrouped  in  an  effort  to  achieve  maximum  accuracy. 

Several  assumptions  concerning  the  data  required 
explication. 

1.  Even  though  the  individual  flight  simulators 

♦ 

simulate  dissimilar  aircraft,  the  systems  themselves 
were  comprised  of  homogeneous  features  and  charac- 
teristics. 

2.  Acquisition  difficulties  encountered  on  the 
systems  within  the  data  base  were  characteristic  of 
difficulties  encountered  in  the  total  simulator 
acquisition  experience. 

3.  Contractor  profit  on  the  various  systems  fell 
within  established  DOD  limits. 

4.  The  system  characteristics  from  the  data  base 
will  also  exist  in  simulator  systems  acquired  in  the 
future. 

5.  The  necessary  data  was  recorded  accurately  on 
the  source  documents. 

Price  Level  Adjustment 

To  remove  the  effects  of  monetary  inflation  from  the 
development  of  the  model  all  total  system  costs/prices  were 
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adjusted.  The  Secretary  of  Defense  Economic  Escalation  Index 
was  the  basis  for  this  adjustment  (23).  The  base  year  (Index 
value  of  100)  utilized  was  1975.  All  price  figures  utilized 
in  model  development  were  expressed  in  terms  of  1975  con- 
stant dollars. 

GENERAL  MODEL  FORMULATION 
General  Considerations 

There  are  numerous  statistical  and  non-statistical 
frameworks  available  for  model  development.  Considerations 
as  to  the  intended  model  use,  data  features  and  analysis 
capacity  available  to  the  researcher  influenced  framework 
selection.  In  this  research  the  suspected  relationship  of 
cost  to  simulator  system  characteristics  pointed  toward  a 
commonly  used  framework. 

Regression  analysis  has  been  a proven  and  accepted 
statistical  procedure.  Prediction  capability  was  one  of  the 
inherent  traits  of  regression  analysis.  This  research 
intended  to  develop  a model  to  facilitate  prediction  of 
price  when  certain  system  characteristics  were  known.  The 
system  characteristics  became  predictors  of  total  system 
cost  when  they  were  input  to  the  regression  analysis  pro- 
cedure. Least  squares  estimation  has  been  the  most  commonly 
used  method  of  regression  analysis  (12:35).  The  least 
squares  method  provided  the  best  linear  unbiased  estimator 
(8:1)  and  was  the  framework  for  the  development  of  this 


parametric  pricing  model.  Certain  assumptions  underlie  the 
regression  method  and  are  enumerated  here  for  clarity. 
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Assumptions 


1.  A linear  model  of  the  general  form 


* - Bo  + B1X1  + B2X2  + 


+ Bp-lXP-l  + E 


was  most  appropriate  for  this  research  due  to  the 
expected  relationship  among  the  variables  involved. 
In  this  general  model  Y was  the  dependent  variable 
(cost),  the  ' s were  independent  variables  (charac- 
teristics), the  B^'s  were  the  unknown  values  to  be 
determined  (parameters)  and  E was  the  error  term. 

2.  The  expected  value  of  cumulative  error  term  (E) 
was  zero. 

3.  All  of  the  individual  error  terms  had  a con- 
stant and  equal  variance. 

4.  The  error  terms  were  statistically  independent. 

5.  The  values  of  the  error  terms  were  normally 
distributed. 

6.  The  number  of  observations  exceeded  the  number 
of  parameters  to  be  estimated. 

7.  The  values  of  the  independent  variables  were 
independent  of  each  other. 

8.  The  values  of  the  independent  variables  were 
observed  without  significant  error  (8:1-3). 
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Computer  Subprogram  Utilization 

Although  the  regression  method  was  amenable  to  manual 
solution,  it  became  extremely  cumbersome  when  more  than  two 
variables  were  considered.  Since  the  cost  of  a flight  simu- 
lator appeared  to  be  dependent  on  numerous  variables,  regres- 
sion analysis  utilizing  digital  computer  support  was  required. 
The  Statistical  Package  for  the  Social  Sciences  (SPSS) 
subprogram.  Stepwise  Multiple  Regression,  was  utilized. 

Stepwise  Multiple  Regression  was  based  on  a common 
method  of  solving  the  system  of  linear  equations  in  multiple 
regression,  that  is,  Gauss-Jordan  elimination.  The  compu- 
tational method  provided  the  information  necessary  to  select 
the  next  variable  brought  into  the  equation.  The  subprogram 
selection  of  variables  occurred  such  that  the  model  provided 
the  best  possible  fit  to  the  data  observations.  After  all 
variables  had  been  evaluated  by  the  subprogram,  an  optimal 
model  was  determined  which  may,  or  may  not,  have  included 
all  of  the  variables.  Variables  appearing  in  the  final  form 
of  the  model  were  those  selected  by  the  subprogram  as  having 
actual  significance.  The  subprogram  also  calculated  the 
parameters  associated  with  independent  variables  included  in 
the  model  (4:180). 

Model  Development  Procedures 

Once  data  had  been  collected  it  was  briefly  analyzed 
with  respect  to  range  of  values.  The  data  was  then  supplied 


to  the  stepwise  subprogram  in  order  of  smallest  range  of 
characteristic  values  to  greatest  range  of  values. 
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Characteristics  which  proved  not  to  be  statistically  signifi- 
cant were  eliminated  from  the  model,  and  the  model  was 
developed  using  the  remaining  variables.  This  build-test 
sequence  was  repeated  until  all  potential  combinations  had 
been  attempted.  When  the  subprogram  had  considered  all 
available  data  values  the  resulting  equation  provided  the 
best  possible  linear  model  for  estimating  flight  simulator 
cost.  The  model  was,  of  course,  limited  by  the  accuracy  and 
quantity  of  data. 

Model  development  was  not  complete  until  the  model 
was  assessed  as  to  its  validity.  Both  internal  and  external 
tests  were  performed  on  the  model  to  determine  the  degree  to 
which  the  model  predicted  cost  accurately. 

PROPOSED  MODEL  VALIDATION 

Statistical  Tests 


Multiple  R test.  The  proportion  of  total  variation  about 
first  unit  simulator  cost  explained  by  the  regression  (com- 
monly known  as  R2 ) had  to  be  significant.  Significance  of  R2 
was  arbitrarily  defined  to  be  0.70,  a value  perceived  to  be 
significant  from  inspection  of  R tables  (15:514).  It  was 
hoped  that  the  R2  of  the  model  would  approach  1.0  as  a 
limit  but  any  value  for  R2  greater  than  0.70  was  acceptable 
for  the  purpose  of  establishing  validity  (26:342). 

F test  on  the  overall  equation.  The  overall  statistical 


significance  of  the  model  was  assessed  by  an  F test. 
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For  the  purpose  of  this  element  of  validation  a 90%  confi- 
dence level  was  established.  If  the  computed  F-ratio 
exceeded  the  critical  (table  value)  F-ratio  at  this  confi- 
dence level,  the  overall  model  was  considered  statistically 
significant  (26:340). 

Student's  "t"  test  on  the  parameters.  The  parameters  were 
assessed  individually  with  "t"  tests  at  the  90%  confidence 
level  to  determine  their  individual  statistical  significances. 
Computed  "t"  values  were  compared  with  critical  (table)  "t" 
values  in  this  assessment  (26:298). 

Standard  error  of  the  estimate.  A measure  of  how  well  the 
model  fit  the  data  was  provided  by  the  standard  error  of  the 
estimate.  The  standard  error  of  the  estimate  computed  from 
the  model  should  have  been  relatively  low  in  comparison  to 
the  standard  deviation  of  the  first  unit  simulator  cost 
data.  In  addition,  the  distribution  of  the  error  terms  for 
each  dat">  point  should  have  been  randomly  scattered  about  the 
estimate  of  the  dependent  variable.  If  this  situation  was 
observed  in  the  output  of  the  stepwise  subprogram,  then  a 
normal  distribution  of  error  terms  could  be  inferred  (20: 
564-565) . 

External  Tests 

Two  systems  and  their  associated  cost  and  charac- 
teristic data  were  to  have  been  chosen  to  be  excluded  from 
the  data  base  used  to  develop  the  model.  Exclusion  of  these 
systems  would  permit  testing  of  the  model's  predictive 
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accuracy  on  them.  That  is,  data  from  the  selected  systems 
was  to  be  supplied  to  the  model  and  the  predicted  first  unit 
costs  were  to  be  compared  to  the  actual  first  unit  costs  of 
these  separate  systems.  To  be  considered  valid  as  a predic- 
tor of  cost  the  model  estimate,  plus  or  minus  one  standard 
error,  should  have  included  the  actual  cost  value.  In  addi- 
tion to  this  straightforward  and  exacting  criteria,  one 
further  analysis  of  the  model  was  to  contribute  to  its 
validation. 

Internal  Tests 

The  independent  variables  were  carefully  scrutinized 
to  insure  that  they  were  logically  related  to  the  dependent 
variable.  That  is,  if  the  relationship  of  the  independent 
and  dependent  variables  appeared  illogical,  the  independent 
variable  was  excluded  from  the  model.  An  example  of  this 
type  of  illogical  relationship  would  have  been  an  independent 
variable  which  should  have  caused  cost  to  increase  but,  in 
fact,  indicated  a cost  decrease.  If  the  reason  for  the 
illogical  relationship  could  be  explained,  the  independent 
variable  was  included,  but  if  explanation  was  not  possible, 
it  was  excluded  (13:340-341).  This  subjective  criteria 
also  contributed  to  model  validity. 

Consistency  to  the  development  process  further 
enhanced  the  model's  validity.  As  previously  noted,  statis- 
tical tests  on  the  model  and  its  parameters  all  utilized 
the  same  level  of  significance.  All  independent  variables 

■1 i i 


received  the  same  treatment  throughout  model  development  and 
only  one  computer  subprogram  was  utilized  in  evaluating  the 
potential  contributions  of  the  independent  variables  to  the 
model . 

MEETING  THE  RESEARCH  OBJECTIVE 

The  research  hypothesis  was  considered  to  be  sup- 
ported if  the  model  met  the  criteria  defined  in  the  preceding 
paragraphs  which  detailed  the  various  statistical,  external 
and  internal  tests. 


Chapter  3 


THE  DATA  BASE 
SOURCES  OF  DATA 

There  was  no  central  repository  for  the  data  upon 
which  this  research  depended.  It  was  assumed  that  contract 
files  would  contain  all  required  data  elements,  but  it  was 
discovered  that  these  files  were  either  inaccessible  or 
incomplete.  It  was  further  learned  that  many  of  the  physi- 
cal and  performance  characteristic  data  elements  did  not 
appear  in  the  contract  files.  Other  data  sources  had  to 
be  sought  and  were  discovered  through  various  inquiries. 

A large  portion  of  the  technical  data  (physical  and 
performance  characteristics)  was  provided  by  personnel  of 
the  Air  Force  Logistics  Command  (AFLC)  at  Ogden  Air  Logis- 
tics Center  (ALC) , Utah.  The  Flight  Simulator  and  Training 
Aids  Branch  of  Ogden  ALC  provided  all  technical  data  for 
those  flight  simulator  systems  which  AFLC  maintains.  Those 
included  the  following  systems:  C-5A,  C-141A,  F-lllA, 

FB-111A,  HH-53  and  A-7D.  The  cost  data  for  these  particular 
systems  was  extracted  from  active  and  inactive  contract 
files  maintained  by  the  Aeronautical  Systems  Division  of 
Air  Force  Systems  Command  at  Wright-Patterson  AFB,  Ohio. 
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The  remaining  systems  in  the  data  base  were  the  F-15, 

T-37B,  T-38A  and  the  Advanced  Simulator  for  Undergraduate 
Pilot  Training  (ASUPT) . All  cost  and  technical  data  per- 
taining  to  the  F-15  flight  simulator  was  originated  by  ASD 
personnel  assigned  to  the  F-15  Systems  Program  Office  (SPO) 
at  Wright-Patterson  AFB , Ohio.  All  data  pertaining  to  the 
T-37B  and  T-38A  systems  was  originated  by  ASD  personnel 
assigned  to  the  Simulator  SPO  at  Wright-Patterson  AFB,  Ohio. 

Data  pertaining  to  the  ASUPT  was  provided  by  ASD  personnel 
assigned  to  the  Human  Resources  Laboratory  (HRL)  at  Wright- 
Patterson  AFB,  Ohio.  Approximately  twenty  (20)  separate 
individuals  from  the  above  listed  offices  responded  to  this 
research  endeavor  with  data  input. 

DATA  COLLECTION  METHODS 

All  data  relevant  to  this  research  was  collected 
and  recorded  manually.  Numerous  lecters,  telephone  con- 
versations and  personal  interviews  were  utilized  to  collect 
the  data.  The  data  collection  process  spanned  an  eight- 
month  period.  It  was  estimated  that  for  every  source  which 
provided  some  portion  of  useful  data,  at  least  seven  addi- 
tional non-productive  sources  were  contacted  during  data 
collection  efforts. 

INDIVIDUAL  DATA  POINTS 

Cost  and  technical  data  was  collected  on  eleven 
different  flight  simulator  systems.  Of  these  eleven  systems, 


two  C-141A  systems  were  included  since  the  two  systems  were 
designed  and  produced  by  separate  independent  contractors. 
Although  some  similarity  was  observed  in  the  two  C-141A 
systems,  the  data  indicated  that  the  systems  were  essen- 
tially different.  Both  the  F-111A  and  FB-111A  systems  were 
included  in  the  data  field  since  the  differences  in  these 
two  systems  exceeded  the  similarities.  For  each  of  these 
eleven  systems  data  on  the  dependent  variable,  cost,  and 
nine  independent  variables,  physical  and  performance  charac- 
teristics, was  collected.  The  technical  data  is  arrayed  and 
presented  in  matrix  form  in  Table  1.  For  a detailed  descrip 
tion  of  the  data  elements  the  reader  should  refer  to  Chapter 
2.  The  cost  data,  which  includes  first  unit  cost,  year  pro- 
cured, index  factor  and  adjusted  cost,  is  similarly  arrayed 
and  presented  in  matrix  form  in  Table  2. 

The  Secretary  of  Defense  Economic  Escalation  Index, 
which  was  used  to  determine  adjusted  first  unit  cost  figures 
was  determined  to  be  the  best  index  available  due  to  its 
wide  application  in  DOD  cost  analysis.  The  reader  is  cau- 
tioned that  all  figures  pertaining  to  cost  are  expressed 
in  constant  1975  dollars  and  that  the  utilization  of  any 
CER  in  this  text  will  require  adjustment  from  1975  to  pres- 
ent or  future  dollars  using  the  escalation  index. 
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DATA  FEATURES 


Strengths 


The  data  was  perceived  to  possess  strength  for  the 


following  reasons: 


1.  There  appeared  to  be  no  bias  on  the  part  of 
those  who  provided  data  to  the  research. 

2.  Numerous  separate  sources  cross-validated  major 
portions  of  the  data. 

3.  All  data  was  expressed  in  quantified  terms. 

4.  Where  numerical  conversions  of  data  was  necessary 
(e.g.  nanoseconds  to  microseconds,  tons  of  cooling 
capacity  to  BTU/hr.)  established  scientific  con- 
version factors  were  available  and  were  applied 
consistently. 


Weaknesses 


Certain  weaknesses  in  the  accumulated  data  were 
perceived  and  are  expressed  below: 

1.  For  some  data  points  different  sources  con- 
flicted as  to  specific  data  values  (e.g.  three 
different  cost  figures  for  the  C-5A,  all  validated 
and  documented;  different  KVA  ratings  and  cooling 
capacity  ratings  for  the  same  system) . 

2.  Some  cost  figures  were  more  "mature"  than 
others.  The  research  recognized  that  a gap  of 
several  years  can  exist  between  acceptance  of  a 
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flight  simulator  and  final  determination  of  its 
actual  cost  after  audit. 

3.  Different  data  sources  applied  unique  definitions 
to  the  data  they  provided  (e.g.  crew  station  weight 
may  or  may  not  have  included  the  weight  of  the  motion 
platform) . 

Overall,  the  strengths  of  the  data  outweighed  the 
weaknesses  and  the  effects  of  some  of  those  perceived  weak- 
nesses were  removed  as  the  data  was  utilized  in  model  devel- 
opment as  discussed  in  Chapter  4. 

Difficulties 

There  was  no  central  or  common  source  of  data.  The 
research  effort  was  bounded  by  both  time  and  data  collection 
method.  It  was  noted  that  no  CER  existed  for  flight  simu- 
lator acquisition  only  because  there  was  no  data  from  which 
a CER  could  be  originated  (1:1).  This  research  endeavor 
found  data  gathering  to  be  both  the  most  demanding  and  time- 
consuming  task.  The  endeavor  further  determined  that  data 
did  exist  in  dispersed  locations,  but  bringing  it  together 
through  manual  means  was  a most  inefficient  operation. 

Data  in  much  greater  detail  was  determined  to  exist  in  the 
records  of  flight  simulator  contractors,  but  contact  with 
these  various  contractors  was  beyond  the  scope  of  this 
research  due  to  time  constraints  and  lack  of  clearance 
authority. 
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was  assembled  possessed  inherent  strengths  and  weaknesses 
which  influenced  the  predictive  power  of  the  CER  it  supported. 


* 


Chapter  4 


r 


PARAMETRIC  COSTING  MODEL  DEVELOPMENT 

Of  the  total  array  of  data  collected  during  the 
research  effort,  three  systems  (i.e.,  ASUPT,  T-37,  T-38) 
were  excluded  from  the  CER  development  process.  The  ASUPT 
system  was  found  to  be  vastly  dissimilar  to  the  other  flight 
simulator  systems  due  to  its  unique  mission  and  design.  The 
ASUPT  was  a single  system  procurement  and  was  designed  for 
contribution  to  research  rather  than  direct  support  of 
operational  flying.  The  T-37  and  T-38  Instrument  Flight 
Simulators  (IFS),  as  well  as  the  ASUPT,  were  configured  as  a 
complex  of  cockpits  rather  than  single  crew  stations.  The 
cockpit  complexes  of  the  IFS  system  shared  common  motion  and 
visual  system  equipment.  These  facts,  coupled  with  other 
minor  dissimilarities,  made  it  difficult  to  separate  the 
components  of  these  three  systems  into  configurations 
similar  to  other  systems  in  the  data  base.  These  three 
systems  were  therefore  excluded  from  CER/model  development 
since  their  potential  positive  contribution  as  additional 
points  in  the  data  base  was  outweighed  by  their  lack  of 
similarity  to  the  other  systems  in  the  data  base.  The 
data  base  used  for  CER  development  therefore  consisted  of 
eight  separate  flight  simulator  systems. 
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INITIAL  ITERATION 

All  data  pertaining  to  the  F-15,  F-lllA,  FB-lllA, 
C-5A,  both  C-141's,  HH-53A  and  A-7D  systems  was  supplied  to 
the  Stepwise  Multiple  Regression  Subprogram.  The  resultant 
equation  included  six  independent  variables.  Three  indepen- 
dent variables  were  excluded  by  the  subprogram  since  their 
contributions  to  explained  variation  were  not  significant. 
Listed  below  are  the  independent  variables  as  they  were 
perceived  on  the  initial  iteration. 


Variables  Included 


Variables 

Parameter  " 

t"  Value 

Sensory  Cues 

23,830,744 

3.934 

Weight 

738 

4.055 

Crew  Positions 

18,324,282 

4.367 

BTU  (Cooling  Cap.) 

82 

3.965 

Deg.  of  Freedom 

6,658,466 

4.045 

KVA  (Elec.  Power  Consump.) 

- 199,082 

3.343 

Constant  (Bq) 

147,870,539 

— 

Overall  F Value  = 13.46 

R2  Value  = 98 

.78% 

F Critical  = 58.2 

"t"  Critical 

= 6.314 

Variables  Excluded 


Computer  Core  Capacity 
Computer  Processing  Speed 
Emergency  Procedures  Simulatable 


The  R2  factor  (percentage  of  variation  explained  by 
the  included  independent  variables)  for  the  above  model  was 
98.78%,  an  extremely  high  value  and  far  in  excess  of  the  70% 
value  established  as  the  R2  criteria.  However,  the  critical 
F statistic  table  value  required  for  statistical  significance 
for  the  above  equation  (with  appropriate  statistical  degrees 
of  freedom)  was  58.2.  The  F value  associated  with  the  six 
variable  model  was  only  13.46.  The  critical  value  of  the 
"t"  statistic  (table  value)  for  all  variables  in  the  model 
was  6.314.  The  preceding  "t"  and  F values  associated  with 
the  derived  model  all  failed  to  exceed  their  corresponding 
table  values  at  the  90%  confidence  level.  This  situation 
was  due  primarily  to  the  relatively  low  number  of  data 
points  (systems) , which  severely  limited  statistical  degrees 
of  freedom  and,  hence,  necessitated  meeting  very  large  F and 
"t"  critical  values.  The  overall  model  and  the  individual 
parameters  failed  to  demonstrate  statistical  significance 
even  though  the  R2  for  the  model  was  exceptionally  high. 

Since  the  model  derived  on  the  initial  iteration  failed  to 
meet  statistical  criteria  further  analysis  of  the  data  was 
performed . 


SECONDARY  DATA  ANALYSIS 

The  potential  existence  of  collinearity  between 
independent  variable  data  was  perceived.  Collinearity 
between  two  variables,  or  multicollinearity  among  several 
variables,  serves  to  weaken  data  from  a statistical 
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standpoint.  That  is,  variables  which  vary  in  a similar  man- 
ner tend  to  mutually  reduce  their  individual  influence  on  the 
predictive  power  of  their  combined  representation.  Therefore, 
a correlation  analysis  was  performed  on  the  data  through  use 
of  a SPSS  subroutine  feature.  It  was  discovered  that  three 
of  the  nine  independent  variables  were  highly  correlated  to 
a majority  of  the  other  variables.  These  three  variables 
which  exhibited  strong  multicollinearity  were: 

Crew  Position 
Sensory  Cues 

KVA  (Electrical  Power  Consumption) 

Therefore,  each  of  the  three  above  named  variables  was 
excluded  from  the  model  development  process.  They  were 
excluded  singly  on  three  separate  iterations  so  that  overall 
changes  could  be  noted  and  assessed.  It  was  discovered  that 
all  three  variables  required  exclusion  to  adequately  reduce 
the  effect  of  multicollinearity  within  the  independent 
variable  data.  With  the  remaining  data,  representing  six 
variables,  further  iteration  was  performed. 

ACTUAL  MODEL  VALIDATION 

Statistical  Tests 

A total  of  eight  separate  systems  with  data  detailing 
the  dependent  variable,  cost,  and  six  independent  variables 
detailing  physical  and  performance  features  comprised  the 
data  base  for  the  refined  CER  development.  The  stepwise 
subprogram  provided  the  following  model  information: 
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Variables 

Parameter 

"t"  Value 

BTU  (Cooling  Cap.) 

70 

3.8601 

Weight 

504 

3.9484 

Deg.  of  Freedom 

7,055,290 

3.4132 

Emerg.  Proc. 

50,623 

3.7054 

Comp.  Speed 

1,470,072 

1.8358 

Comp.  Core 

- 36,452 

1.1446 

Constant  (Bq) 

50,  529,969 

Overall  F Value  = 5.85 

R2 

Value  = 97.23% 

F Critical  =58.2  "t"  Critical  = 6.314 

The  overall  model  F value  failed  to  exceed  the 
critical  value  for  F at  the  90%  confidence  level.  None  of 
the  "t”  values  for  the  individual  parameters  exceeded  "t" 
critical.  Therefore,  this  particular  form  of  the  model 
failed  to  meet  the  statistical  criteria  and  was  therefore 
deemed  unsatisfactory  as  a CER. 

Since  the  form  of  the  model  based  on  six  parameters 
was  not  statistically  adequate,  the  model  form  based  on  five 
parameters  was  considered.  Below  are  listed  the  salient 
features  of  the  five  parameter  model. 
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Variable 

Parameter 

"t"  Value 

BTU  (Cooling  Cap.) 

58 

3.6688 

Weight 

475 

3.5313 

Deg.  of  Freedom 

5,637,063 

3.1654 

Emerg.  Proc. 

43,187 

3.3407 

Comp.  Speed 

995,890 

1.3491 

Constant  (B  ) 
o 

41,701,176 

Overall  F Value  = 5.84 

R2 

Value  = 93.59% 

F Critical  = 9.29 

"t 

" Critical  = 2.92 

The  overall  F value  for  the  five  parameter  model 
failed  to  exceed  the  critical  F value  but  the  difference 
was  much  less  than  was  observed  in  the  six  parameter  model. 
The  individual  "t"  statistics  for  the  first  four  parameters 
were  significant  while  only  the  last  parameter,  computer 
processing  speed,  failed  in  the  test  of  significance.  A 
reduction  in  the  R2  value  was  noted  in  comparison  to  the 
six  parameter  model  as  might  have  been  expected.  These 
results  were  described  as  being  typical  during  the  dis- 
cussion of  the  multiple  linear  regression  technique  in 
Chapter  2.  The  five  parameter  model  partially  passed  the 
statistical  test  criteria  and  indicated  an  assessment  of  a 
four  parameter  model  would  demonstrate  further  refinement. 

The  four  parameter  model  met  all  relevant  statis- 
tical test  criteria  as  the  following  table  indicates. 


Variable 

Parameter 

"t"  Vali 

BTU  (Cooling  Cap.) 

50 

3.0223 

Weight 

369 

2.9938 

Deg.  of  Freedom 

4,003,544 

2.7157 

Emerg.  Proc. 

35,232 

2.7126 

Constant  (B  ) 
o 

- 28,274,648 

Overall  F Value  = 5.38  R2  Value  = 87.76% 


F Critical  = 5.34  "t"  Critical  = 2.35 

The  R2  value  of  87.76%  exceeded  the  criteria  value  of  70%. 
The  F value  of  5.38  exceeded  the  critical  F value  of  5.34 
at  90%  confidence  level.  All  "t"  values  exceeded  the  "t" 
critical  value  of  2.35  at  90%  confidence  level.  The  four 
parameter  model  was  therefore  found  to  be  statistically 
significant. 

Standard  Error  of  the  Estimate 

It  was  required  that  the  standard  error  of  the 
estimate  computed  from  the  model  be  relatively  low  in  com- 
parison to  the  standard  deviation  of  the  first  unit 
simulator  cost  data.  The  standard  error  of  the  estimate 
and  standard  deviation  were: 

Standard  error  — $2,728,014.17 
Standard  deviation  — $9,055,186.76 
The  standard  deviation  was  calculated  from  the  cost  data 
appearing  in  Table  2 and  the  standard  error  was  calculated 
by  the  SPSS  program  for  the  four  parameter  model.  The 
standard  error  of  the  estimate  was  relatively  low  in 
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comparison  to  the  standard  deviation  of  the  cost  data.  The 
standard  deviation  was  in  fact  more  than  three  times  greater 
than  the  standard  error. 

The  distribution  of  the  error  terms  for  each  data 
point  was  to  depict  a random  scatter  about  the  estimate  of 
the  dependent  variable.  This  random  scatter,  if  observed, 
would  have  indicated  a normal  distribution  of  the  error 
terms.  A normal  distribution  of  error  terms  was  necessary 
to  permit  the  assumption  of  no  bias  in  the  estimate.  Such 
a distribution  of  error  terms  was  observed  and  is  presented 
in  Table  3.  Additionally,  Figure  1 depicts  a plot  of  these 
residuals  (error  terms)  as  a percentage  of  actual  cost 
values.  Overall,  the  assumption  of  a normal  distribution 
of  error  terms  was  valid  and  an  unbiased  estimate  was 
inferred.  Having  developed  a four  parameter  model  that  met 
the  established  statistical  criteria,  the  remainder  of  the 
model  validation  process  ensued. 

External  Test 

It  was  intended  that  the  developed  model  would  be 
applied  to  a known  system  and  its  predictive  ability  assessed 
in  that  manner.  Due  to  the  limited  amount  of  data  however, 
there  was  no  system  available  to  which  the  model  could  be 
realistically  applied.  The  three  systems  excluded  from  the 
data  base  for  CER  development  (T-37,  T-38  and  ASUPT)  were 
so  dissimilar  to  the  remaining  systems  that  application  of 
the  model  to  those  three  was  deemed  inappropriate.  Ultimate 
validation  of  the  model  through  external  test  on  similar 
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flight  simulator  systems  was  left  to  those  who  might  benefit 
from  future  use  of  the  CER.  It  was  noted  however  that  all 
systems  utilized  in  model  development  could  be  predicted  by 
the  model  as  to  their  actual  costs  plus  or  minus  one  stan- 
dard error  with  only  one  exception. 

Internal  Test 

The  model  was  assessed  as  to  its  internal  form.  All 
variables  were  found  to  be  logically  related.  That  is,  the 
coefficients  of  the  independent  variables  were  positive 
which  indicated  a direct  relationship  to  cost.  Statistical 
methodology  was  consistent  throughout  model  development  and 
all  variables  received  equal  treatment. 

It  was  necessary  to  assume  that  the  model  error 
term,  E,  would  be  zero.  This  was,  in  effect,  the  case. 

The  sum  of  the  residual  values  (individual  error  terms) 
depicted  in  Table  3 was  a minus  $4.03.  This  value  approxi- 
mated zero  quite  satisfactorily  considering  the  relative 
magnitudes  of  the  residual  values  which  contributed  to  the 
sum.  The  assumption  of  a zero  value  error  term  in  the  model 
was  therefore  supported. 

CER/Parametric  Model 

The  parametric  model,  or  CER,  developed  is  expressed 
below  in  equation  form. 


Y = -28,274,648.96  + 50.19XL  + 369.26X2  + 4 , 003 , 544 . 50X3 
+ 35,232.25X4  + E. 
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In  this  model  the  variables  were: 

Y - System  first  unit  cost 

- System  cooling  capabity  in  BTU/hour 

- System  weight  in  pounds 

X^  - System  degrees  of  freedom  for  the  motion  platform 
X^  - System  emergency  procedures/malfunctions  simulatable 
E - Error  term  for  the  model 

Cost,  expressed  in  1975  constant  dollars,  was  the  dependent 
variable  of  the  above  model.  The  independent  variables 
appearing  in  the  above  model  were  those  found  to  be  statis- 
tically significant  as  predictors  of  cost.  The  error  term, 

E,  can  be  considered  as  having  zero  value. 

SUMMARY 

The  CER  was  developed  in  accordance  with  the  criteria 
established  in  Chapter  2.  All  criteria  tests  were  passed 
by  the  CER  with  the  exception  of  the  external  test  criteria 
which  could  not  feasibly  be  accomplished.  Therefore,  the 
research  hypothesis  was  considered  supported. 


Chapter  5 

MODEL  USAGE  AND  APPLICATION 
PREDICTIVE  RANGE 

The  CER  should  be  used  to  predict  the  first  unit 
costs  of  flight  simulator  systems.  Data  required  to  utilize 
the  CER  includes  known  or  predicted  quantitative  values  for 
system  cooling  capacity,  weight,  motion  platform  degrees  of 
freedom  and  emergency  procedures/malfunctions  simulatable. 
Extrapolation  of  data  values  and  associated  cost  predictions 
beyond  the  values  in  the  data  base  will  necessarily  cause 
those  predictions  to  lose  accuracy  (12:56).  The  greater  the 
extent  of  extrapolation  beyond  data  parameters  the  greater 
will  be  the  decrease  in  predictive  accuracy  of  the  CER. 

To  insure  that  the  CER  possesses  a data  base  which 
will  permit  a realistic  predictive  range  at  any  point  in 
time,  constant  update  of  the  CER  is  mandatory.  Without 
update  of  the  data  base  advancing  technology  will  render 
the  CER  unuseable  due  to  changing  data  parameters. 

CER  UPDATE  AND  MAINTENANCE 

The  Armed  Services  Procurement  Manual  (ASPM)  encour- 
ages those  activities  which  are  the  primary  users  of  a CER 
to  maintain  responsibility  for  update  and  maintenance  of  the 
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CER  (23:2D35).  The  CER  which  is  the  product  of  this  research 
should  be  maintained  by  the  primary  flight  simulator  acquisi- 
tion office  in  the  U.S.  Air  Force,  The  Simulator  Systems 
Program  Office  of  the  Aeronautical  Systems  Division,  Air 
Force  Systems  Command.  That  office  should  have  primary 
responsibility  for  continually  guaranteeing  CER  data  base 
currency  as  well  as  providing  the  continually  updated  CER 
to  other  offices  which  might  benefit  from  its  use. 

CER  maintenance  and  update  require  on-going  data 
collection,  continual  reassessment  of  variables,  search 
for  more  powerful  variables,  application  of  evolving  sta- 
tistical procedures  and  any  other  practices  which  might 
contribute  to  further  CER  refinement. 

This  research  endeavored  to  conform  to  accepted  CER 
development  procedures.  The  decision  rules  which  were 
enumerated  in  preceding  chapters  were  consistent  with  both 
descriptive  and  inferential  statistical  procedures  as  well 
as  established  model  development  practices.  It  is  noted 
however  that  no  model  perfectly  describes  reality  except 
by  chance.  Utilizers  of  the  CER  this  research  has  devel- 
oped should  be  mindful  of  the  inherent  limitations  of  any 
statistical  model.  CER  utilization  should  be  but  one 
element  of  a total  cost  estimation  approach. 

As  a CER  data  base  expands,  the  potential  for  CER 
refinement  exists.  A CER  approaches  predictive  perfection 
only  as  the  data  base  from  which  it  is  derived  approaches 
the  status  of  being  collectively  exhaustive. 
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Chapter  6 


SUMMARY 

ASSUMPTIONS 

This  research  was  predicated  on  important  assumptions 
and  limitations.  They  are  consolidated  and  listed  here  for 
clarity. 

1.  The  system  for  which  the  estimate  is  being  made 
must  be  closely  related  to  the  systems  used  in  the 
data  base  to  define  the  relationship,  and  it  must  be 
assumed  that  the  defined  relationship  still  exists. 

2.  The  data  in  the  data  base  is  accurate  in  its 
raw  form,  i.e.,  the  data  base  was  recorded  in  the 
contract  files  without  error. 

3.  Separate  simulator  system  characteristics  are 
adequately  homogeneous  for  aggregation  into  specific 
identifiable  categories. 

4.  Acquisition  difficulties  encountered  on  the 
systems  within  the  data  base  are  characteristic  of 
difficulties  encountered  in  the  total  simulator 
acquisition  experience. 

5.  Contractor  profit  on  the  various  systems  con- 
forms to  established  DOD  limits. 

6.  A linear  model  is  most  appropriate. 
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7.  The  expected  value  of  the  error  term  (E)  is  zero. 

8.  All  of  the  error  terms  have  a constant  and  equal 
variance . 

9.  The  error  terms  are  statistically  independent. 

10.  The  values  of  the  error  terms  are  normally 
distributed . 

11.  The  number  of  observations  exceeds  the  number 
of  parameters  to  be  estimated. 

12.  The  values  of  the  independent  variables  must 
be  independent  of  each  other. 

LIMITATIONS 

1.  Accuracy  of  the  cost  estimate  is  limited  by  the 
fact  that  physical  and  performance  characteristics 
do  not  represent  all  of  the  reasons  for  variance  in 
cost . 

2.  When  extrapolating  beyond  the  values  of  data  in 
the  data  base,  confidence  in  the  accuracy  of  the 
estimate  decreases. 

3.  Predictive  strength  of  the  model  is  limited  by 
the  quantity  and  quality  of  the  data  base  from  which 
the  model  is  developed. 

RECOMMENDATIONS 


The  purpose  of  this  research  effort  was  to  develop 
a parametric  costing  model,  or  CER,  for  flight  simulator 
acquisition.  Extensive  research  was  conducted  to  acquire 


47 


the  necessary  background  information  as  well  as  to  describe 
the  development  methodology  utilized.  Statistical  methods 
were  explicitly  detailed.  Data  utilized  in  model  develop- 
ment was  depicted  as  to  source,  type  and  salient  features. 
Data  collection  methods  were  described.  CER  development 
was  chronologically  reported  according  to  the  established 
methodology.  The  CER,  or  parametric  costing  model,  was 
presented  along  with  pertinent  comments  concerning  its 
utilization  and  applicability. 

Ultimate  validation  of  the  CER  is  beyond  the  scope 
of  this  research.  Data  deficiency  prevented  a meaningful 
test  of  the  CER's  predictive  accuracy  within  the  context 
of  this  research.  Future  application  of  the  CER  to  real 
world  simulator  acquisition  processes  will  reveal  the  CER's 
worth.  Even  if  the  CER,  in  its  present  form,  cannot  be 
validated  through  application  it  should  not  be  abandoned 
as  a management  tool.  The  CER  should  be  updated  and 
refined  continually  with  emerging  data  so  that  its  value 
to  the  simulator  acquisition  process  is  further  enhanced. 

It  is  recommended  that  the  office  which  maintains 
this  CER  establish  procedures  for  data  collection  and  data 
update.  This  research  effort  was  limited  by  both  data 
availability  and  accessibility.  To  further  refine  this 
CER/model  will  require  additional  validated  data  and  removal 
of  outmoded  data.  Contract  specifications  for  contractor 
provided  data  should  include  cost  and  physical/performance 
characteristics  data  similar  to  the  data  from  which  this  CER 
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was  developed.  This  method  would  guarantee  continuing  data 
availability  and  provide  for  update  and  refinement  of  the 
CER/model . 

This  initial  research  effort  in  the  area  of  para- 
metric cost  estimation  for  flight  simulators  utilized 
aggregated  descriptions  of  flight  simulator  cost,  physical 
and  performance  characteristics.  Future  research  is  deemed 
feasible  at  a less  aggregated  level  of  description.  That 
is,  the  CER  developed  herein  may  well  be  complemented  by 
CER's  for  flight  simulator  subsystems  and  components. 

This  possibility  rests  on  the  availability  of  data  at  the 
subsystem  level.  Overall,  the  necessity  to  collect  and 
maintain  these  types  of  data  has  been  shown  to  be  critical 
to  successful  and  expedient  CER/model  development'. 
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